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THIS PAPER DESCRIBES THE RULEG PROGRAM SYSTEM, WHICH IS 
BASED ON THE PREMISE THAT THE VERBAL STATEMENTS OF A PROGRAM 
CAN BE CLASSIFIED INTO "RU'S" (RULES) AND "EG'S" (EXAMPLES). 
ALL RU'S ARE STATEMENTS OF SOME GENERALITY, FROM WHICH 
SUBSTITUTION INSTANCES, OR EG'S, CAN BE OBTAINED. 12 STEPS 
FOR THE CONSTRUCTION OF RULEG PROGRAMS ARE LISTED— SPECIFY 
THE CRITERION BEHAVIOR, WRITE DOWN MANY RU'S, COLLECT 



STIMULUS SUPPORT, ORDER THE RU'S, MAKE A RU MATRIX, CONSTRUCT 



EG'S, NUMBER THE MATRIX CELLS, ASSEMBLE RU'S AND EG'S INTO 
FRAMES, AND THE FRAMES INTO A PROGRAM, PRE-TEST AND THEN 
REVISE THE PROGRAM UNTIL RELIABLE CRITERION BEHAVIOR IS 
ACHIEVED. (LH) 
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PjREFACB 



It is now generally realized that the success of a teaching 
machine depends greatly upon the material used in it and that the 
development of principles and techniques for programming this material 
is im important area for investigation. At the present time work in 
the field. f Oteaching Machines and Progranmed learning hy A. A. 
Lumstlaine and R. Glaser, i960. National Education Association) has 
suggested certain rudimentary techniques.^ The eventual goal# however, 
is the develpBoent of a technology of programming based upon the 
science of learning and upon the results of detailed experimental 
tryout. The present report describes an approach toward a systematic 
procedure for the construction of programmed learning sequences for 
verbal materials. Esq^erimental study with a program prepared accord- 
ing to the system described in this report is presented in a separate 
project publication (An Investigation of ^ Teaching Machine” Vari a b les 
Uajng Lpar"<ng Programs in iSymbolic logic, James L. Evans, In Press). 



^A comprehensive review of the present notions undw^i^ programming 
is also presented in a separate project report, (^incljgles and 
Problems in ^ Preparation of Programoed Learning ^quences, 

Robert Glaser, In liess).” 
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Introduction to the RUIEQ System^ 



!Ihe RULBG system is based on the premise that the verbal subject 
material ^Ich appears in a program can be classified into two classes 
of statements \diich we will call Ru’s (for ’’rules”) and EC's (for 
”exan5>les”). Learning to identify RU's and EG's is itself a problem 
in concept formation* Therefore^ we will deal with it as with any 
other generalization -and-dlscrlmlnatlon problem* !!Diat Is^ we will 
attempt to give a range of both RU’s and BG’s together, all properly 
labeled, so that we can learn to distinguish between the two concepts, 
althou^ we may find at times that the two concepts are not mutually 
exclusive* This that the concept of the RU will, on occasion, 

shade into the concept of the EG. OMs ’’relativity” of RU’s and EG’ s 
in which the same verbal or synibollc statement may sometimes be both 
a RU and an EG has so far not caused much difficulty. On the contrary, 
it hftf * proved useful in the construction of programs* Instances of 
statements \diich can be both a RU and an EG will be considered shortly* 
We have not yet discovered a completely satisfactory definition 
of a Ru, nor are we convinced at this stage that we need one* Our 
strategy will be to exemplify as widely as possible statements \dilch 
we have classified as RU's. A Ru may be a large number of things. 

It may be a definition, operational or otherwise. It may be a math- 
ematical formula. It may be an empirical law. It may be a principle , 
axiom, postulate, or hypothesis from any area of knowledge. But the 
invariant feature of all RU’s is that they are aU statements of some 
generality, from which substitution-instances can be obtained. These 
substitution-instances are called BG’s* An EG may be a large nuniber of 
things. It may be a description of a physical event* It may be a 
theorem or deduction of any sort. It may be a statement of a relation- 
ship obtaining between specific objects, whether the objects are physi- 
cal or conceptual. But the invariant feature of all EG’s is that they 
are all statements of some s pecificity, derived frcan more generalized 

RU’s. 
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, ^Since any set of principles of programming is itself verbal 
material, such principles could no doubt be best presented in program | 

form. But to escape getting caught in an infinite regress Involving 
programming programs on how to program, this paper will appear in 
conventional form* 
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!Ilie clearest Instance of BU's and their corresponding EG's come 
from mathematics. For exasqple^ the algebraic " a h- b a b jf;a'S is 
a W, iMhich sunmarizes cooipactly an (infinite) nuznber of substitution- 
instances^ or EG's, such as "7 2 a 2 + 7 ” find ” 3 .^ + 8.6 * 8.6 + 3 »^"« 

Again^ .the- statement "Unsupported objects will fall toward the earth" 
is a RU/ ^diile "If I release my pencil it will fall" would be an 
as would "If I release my book^ it will fall". From optics we have 
the HU that "The angle of incidence is equal to the angle of reflection". 

vO 



To exemplify^ we create EG's such as "If the angle of incidence is 30 , 
the angle of reflection is equal to 30°". From psychology we have 
'•Withold reinforcement from an organism and extinction occurs" for 
the BU^ and "If I stop giving the rat pellets^ he will extinguish 
the appropriate response" as an EG for the HU. Of course^ all of 
these HU'S (including the mathematical ones which are more precise) 
would need qualification to become acceptable^ but the point should 
be clear that RU's involve some generality, while EG's involve speci- 
ficity, albeit a relative specificity. That is, "3 + 2*2+ 3 ” Is ia 
turn a RU for such EG's as "3 stones + 2 stones = 2 stones + 3 stones" 
and "3 mammoths + 2 mammoths * 2 mammoths + 3 mammoths". Indeed, the 
early man ^o induced the "3+2*2+ 3 ” RU from these EG's for the 
first time was probably staggered at the profundity of his own in- 
sight. Several eons passed before a generalization was made to the 
"a + b » b + a" RU, making a new EG out of an old RU. And it remained 
for a modern group theorist to produce that ultimate (?) RU "aob * boa" 
in ^dilch nether the objects nor the operator (o) are specified, and 
in which "a + b * b + a" becomes a mere EG. Other examples of RU's 
fljid EGr's could be drawn from the physical sciences, to demonstrate the 
relativity and inter -dependence of RU's and EG's. With this RU-EG con- 
cept in mind, let us now see how it is related to the preparation of 
programmed le a r n**^*^ sequehces and proceed to the steps in the construc- 
tion of a program. 



aiEP 1. Specify the criterion behavior . At this step the programmer 
must attempt to outline, as precisely as possible, both the responses 
he wants from the student at the end of the program, and tfae stimuli 
or cures in the presence of idxlch the student will be expected to make 
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these responses. Ihls is the tine when questions such as the fol- 
lowing must be asked and answered. If the student is studying 
statistics, will he have to produce the formula for the standard 
deviation on his own, or will he always have a book available? 

Is he being prepared to write a short essay comparing two statisti- 
cal tests or will he be called upon to take a multiple -choice test 
at the end of the program? In psychology, will we be satisfied if 
he can identify and label examples of regression or spontaneous 
recovery, or will he be asked to recommend ways of preventing the 
occurrence of these phenomena? In language, do we want to build 
In a complete reading, writing, and speaking knowledge? Or would 
we be satisfied If the student can merely read the technical lit- 
erature in his. chosen field with the aid of a dictionary? In 
mathematics, do we want the subject to be able to solve problems, 
or to prove theorems, or both? With vhat stimulus -support may the 
student provide himself while we are assessing his criterion be- 
havior? Can he use a book? Another student? The instructor? 

His notes? A table of integrals? The construction and form of the 
program will differ radically as a function of the criterion behavior 
chosen. Now is the time to be both realistic and specific in stating 
\diat the objectives of the programmed course are. 



STSP 2. \frlte down all the BU*s you can think of . Do this first in 
the absence of any support from books, notes, charts, or advice from 
esgperts. These will all be available later. At this point you want 
to come up with as much on yoinr own as you can. If you look at a 
familiar text first, you may be off on a response chain \d^ch will 
bypass a number of useful rules which you otherwise could have pro- 
duced. In other words, you may fall too strongly under the control 
of some previously learned verbal habits in that area. If you are a 
subject-matter expei't, this step may produce all the RU‘s which you 
will need for your program; if you ere trying to program a new course 
as you learn it yourself, it may produce only a few RU's. In any event, 
you should try to get out as much verbal behavior on the topic as you 
can before you begin to utilize external stimulus supports such as 
textbooks. Write down each RU on a separate index card. 
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S!EEP 3» Collect all the stimulus support you can In the form of 
textSs notes, advice^ and so on . Examine these systematically for 
the RU's Tdiich you vish to build into your students' repertory, ^ite 
these RU's on index cards also. If you find EC's only, induce the 
RU yourself and vrite it down. (At this step as well as in many of 
the preceding and following steps, do not be overly critical of your 
efforts. She ultimate critic is the student; he will criticize your 
program by his ability or inability to perform as desired. Do not 
try to guess Aether your program is good or bad. Too detailed 
criticism at this time will slow programmers down and will take 
longer to get in the hands of a student. Once the program is out, 
you can get feedback on it and revise it.) 

STEP Usike some prelim'tnn.Yy ordering of RU's . RU's have to be 
presented in some order, but the conventional order in which they 
appear in texts is not necessarily the best order. You have put 
RU's on separate cards to make it easier for you to experiment with 
orders of presentation. A number of ordering relationships are use- 
ful here, such as complexity (introduce simpler RU's first), chronology 
(ordering RU's in time, as in perhaps a history program), spatiality 
(ordering RU's in space, as in perhaps a geography program), and, in 
particular, dependence on other RU's. To illustrate the last re- 
lationship you would introduce RU's about the mean in a statistics 
program before you introduce RU's about the variance, since the defini- 
tion of the variance involves the definition of the mean. Watch for 
omitted RU's; don't force your student to deal with unfamiliar terms, 
or old terms which may need a bit of refreshing. You can anticipate 
this by a careful consideration of the RU's and the behaviors involved. 

STEP 5 . Make a RU matrix . List all your RU's vertically like this; 

RU 1 

RU 2 
RU 3 
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!Rien^ list the some HU's horizontally like this: 



HU 1 


HU 2 


HU 3 


• • • 


HU 1 








m 2 








HU 3 








• 

# 

• 

1 


1 







GQie use of a matrix is based on the fact that the hallmark of the 
expert in a subject-matter is his ability to interrelate the concepts 
in his field. The HU matrix permits a systematic method for examining 
the HU's of a subject singly^ in ordered pairs^ triplets > and so on. 
HU's can be checked for similarities^ differences^ possible confusion^ 
or any of a host of possible intra-verbal connections. The matrix is 
also not without heuristic value ^ in that it may prompt comparisons 
or relationships not previously considered ("How are the laws of 
thennodynamlcs related to the laws of optics?" "How is what we know 
about bread mold related to bacterial growth?"). 

The upper-left corner of the matrix is reserved for operators 
for inter -relating the HU's. A very general operator is the relator 
operator. Consider a matrix like this: 



relaxe; 


HU 1 


HU 2 


HU 3 


BU 1 


1 


2 


3 


HU 2 


4 


5 


6 


HU 3 

t 


7 

! 


8 


9 



As we consider cell ‘Hk, for exacple^ we would ask "How is HU 2 re- 
lated to HU 1?'^ This might require a different handling then cell 
#2, which asks, "How is HU 1 related to HU 2?" If the order of re- 
lationship is of no consequence, symmetrically placed cells, such as 
2 end 4, 3 ond 6 3 redundant. If ordering does matter, the 

complete matrix permits examination nf every ordered pair. Such order- 
ed pairs may be ndded to the axes as new opeponts (row and column 
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entrlee} to permit expansion of the matrix to ordered triplets ^ 
quadruplets^ etc* For example^ ve could first relate pressure to 

4 

vol\ane in the "perfect” gas equation, then put this relationship on 
the exes of the matrix, and have a nev cell relating pressure and 
volume to temperature. 

QUhe major diagonal relates each rule to Itself, and we have 
found it useful to reserve this diagonal as definition cells . We 
"relate a HU to Itself” hy defining the HU in terms of some previous 
behavior tdiich we can safely assume exists at sufficient strength in 
the student to mahe the definitions meaningful. 

If we use another useful operator, fdilch we mi^t term the. 
discriminator operator, ve might get a matrix like this: 



SISCBBOlAXtffi 


HU 1 


i HU 2 


HU 1 


1 


f 

2 


BU 2 

i 


3 

\ 1 


k 



Cell #2 would permit us to ask "How is HU 1 different from HU 2?” 

OMs may set the stage for discrimination training if the HU's have 
formal or thematic similarities which may later confuse the student. 

fflCEP 6. Use the example operator in the HU matrix to construct 
examples for the program . Ihis is a critical feature of the RUIB3 
system. !Ihe EC's which ore constructed must meet a nimiber of criteria. 
Ihese are: 

1. An adequate number of EC's must be generated. It is mainly 
through EC's that the student will Interact with the subject-matter. 
EC's must be generated not only to exemplify a HU initially, but to 
provide 2nter practice and review for that HU. 

2. The full spectxm of EC's for a given HU must be considered. 
This means considering special cases, limiting cases, trivial cases, 
examples with inadequate Information, and examples with redundant 
information. As a good rule of t;husib, the first EC for a given HU 
should be the simplest possible non-triviaX eigample . For Instance, 
for a student's first example of the computation of a variance, use 
an W of 2, with simple integer scores idilch produce a mean which is 








also aa Integer. For example^ use 3 ^ idilch give a mean of 

Two scores are adequate to illustrate the hasic operations involved. 

An EG ^th an N of 1 is an important special case, but it can be saved 
for later. Also, EG's in \diich all the scores are the same, such as 
4 , 4 , 4 , and 4 , should be reserved for later. Decimal or fractional 
scores (e.g., 7«319 or 8 lll6), if ^ised on the initial EG, would 
divide the student *s attention between learning the operation and 
fussing with the arithmetic at a crucial stage in his learning. 

Leading off with a simple non-trivial EG for a RU is the sine qua non 
of good programming. Any temptation to be complex or devious should 
be resisted at this time; there will be opportunity for this when the 
student's behavior has been adequately strengthened. Complex "test 
items" come later. 

3* As pointed out before, to insure adequate generalization of 
a RU the attached EG's must be as diverse as possible and still exestplify 
that RU. Also, to insiire adequate discrimination bet^r^en RU's, EG's 
must be selected \diich resemble each other as nearly as possible ^dille 
still exemplifying their respective RU's. 

STEP 7. Number the cells of the original RU matrix to indicate the 
order in which the RU's are to be presented and exemplified . It is 
often useful to start off "down the diagonal", utilizing the major 
diagonal (the intersection of each RU with itself) as a definition 
frame * After that, you can decide "vdiat RU's you want to compare and 
contrast, what pairs of RU's you wish to omit (be cautious of labeling 
RU's as being "unrelated"; like a careful driver, take all "inter- 
sections" seriously). Number in sequence all cells with which you 
wish to deal. Uhls numbered RU matrix will be a useful step in the 
assembly of the final program. 

STEP 8* Begin assembling the RU's and EG's into frames * A frame 
consists of the stimulus material available at auy given moment which 
demands one or more responses on the part of the student. The student 
makes his indicated response, and then receives immediate feedback as 
to the correctness or incorrectness of his response. Frames are con- 
structed by judicious selection and combination of RU's and EG's from 
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the population of RU’s and EC's 'which have heen constructed. Hb 
provide for responses on "the part of the student^ words ^ phrases^ 
numbers, symbols and so on are omitted from certain RU's and certain 
EGr*s. Such omissions are indicated wl'th a rectangular box, like this: 

I I . For instance, we could give an incomplete RU in this 

manner: Ohe angle of incidence is equal to the angle of 1 I 

where "reflection” is to be provided by the student. As an example 
of an incomplete EG involving n\mbers, *we might have: 

8 + 3 = 3 + r 

with *8* being the appropriate response, ©le choice of the response to 
be made by the student is obviously an important matter. It often is 
easier for stylistic reasons to cmait certain words ffom a frame than 
others; however, it is necessary to consider Just \diat the response re- 
quired by the student can accomplish toward producing the criterion 
behavior. Does the response review or strengthen the responses 
elicited in part frames? Does it help set him for the next s'tep? 
Does it "test" \diat he should have learned previously? 

To designate the inconplete RU»s and EG's, -we add the symbol "^" 
(read tilde) over the symbols for rules and examples, like this: RU 

(RU tilde) and EG (EG tilde). For terminal situations in \diich cri- 
terion behavior is being called for in the presence of minitnfl .1 stimulus 
support, use two tildes over the RU and EG symbols, like this: RU, EG. 

Ad. exacple of a RU would be: 

©le machine formula for the Pearson-product-mcmient correlation 

coefficient is 1 | . 

No O'tiier hints or aids 'would be available to the student. An example 
of an EG would be: 



©le variance of -the following scores: 3, 7, 4, 2, 9 

r “! • 



Again, no other RU*s or BG*s which might serve as prompts must be present, 

>i> 55^ 

or we would have an EG, not an EG. 

We 'Will now present a list of frame -types which we have found to 
be most useful in program cons-truction, as well as a rationale for the 
use of each frame -type. 
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1. RU EG ■» EG * We have found this frame-type to be the method 
par excellence for the economical introduction of a new RU. The 
RU + EG -f K can have the student working an exaaiple of a brand 
new rule on the very first frame in idiich he is e^qposed to it. 

By giving b'tw an explicitly stated RU, and one (or more) carefully 
chosen BG's before calling for a response by means of an EG, a 
powerful prompt is set up which makes an error in responding most 
unlikely. To illustrate a RU + EG + EG frame: 



TO multiply by ICX), move the decimal point two places to the 
right. (RU) For example, 100 x ^•Qh3 = 284.3. (EG) Also, 
100 X 7.374 = I 



]. (EG) 



This is a critical phase in the development of a program. It seems 
best not to introduce more th8m one RU at a time, or to introduce 
complex RU's too early, or to ignore consideration of the "simplest 
possible non-trivial example". (The first exanple in many texts in 
an effort to be realistic is often much too complicated. ) Work the 
student up to conplicated and realistic examples. If 20 redundant 
frames are needed to present a RU adequately, then use 20 redundant 
ffames, and don't worry about the extra space you use. If someone 
a^s, "Can't you omit that example? You have used one very much like 
it before," let the student's performance decide Aether you will 
delete it or not. The primary consideration is that prerequisite 
behavior must be adequately strengthened before the student can 
proceed. (The usual introductory text or lecture may not effectively 

provide for performing this function.) 

We have found then that RU + EG + EG (in that .Order) iS‘ the most 
effective method of presenting an initial frame for a new HU. How- 



ever, other combinations are certainly possible. For exanple, why not 
RU + EG? We have found that at least one completely worked-out 



example is necessary to permit the working of the first EG with ease 
and efficiency. Why not EG + EG? This would be an analogy frame. The 
chief characteristic of an analogy is that one must induce the RU from 
the first EG, and then apply it to the EG. Rather than run the risk 
of having the st\adent induce an incorrect RU, it seems preferable to 
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Tamillsr vords with new meanings to he attached, o.g., population 
In statistics. Students often are slow in adding such words to their 
active vocabulary. If a TV wcrd is defined in one frame, and then 
called for. after subsequent frames, it has been our experience that 
the student will often fall to cc^ up with the word, or will come 
up with the word only with difficulty. It has proved good technique 
to force attention to the TV tem^ by calling for them e:q>llcitly soon 
ef^ter they are introduced. To illustrate: 

The Commutation Law permits the interchance of members con- 
nected by a sign. (BU) So if you found a case \diere 
two numbers had been switched about a sign you could 
be sure that the I i had been used. (RU) 

3. RU + EG . In late frames stimulus support can start to be 
withdrawn by giving only the RU before giving the EG. Likewise the 
previously mentioned '’analogy" frame (EG + EG) represented a weaker 
prompt for the EG than does RU + EG + EG. 

4. EG + RU . An induction frame \dilch can be used more safely 
when the student has the RU at some strength. 

yv> 

3 . RUj^ -I* RUg. . Such frames can be used to compare and contrast 
two different RU's as part of the discrimination training involving 
the RU's. This can obviously be extended to RU^ + + RU- and so on. 

6. For an. contrasting t™ 

different RU's, or for demonstrating how two RU's can be applied in 

ii 

turn. Again the extension to EG, + EG^ + EG-, etc., is obvious. 

0^ J- & o 

7« The terminal frame of a series in which criterion be- 

havior is performed in the presence of minimal cues. For example, we 
mi£^t ask the student to "Compute an analysis of variance on the fol- 
lowing three groups of score s..." or "Describe the acetylcholine cycle." 

o. Althou^ the EG frame is more basic, in the sense that 

to respond appropriately to an EG, the student must have command of 



I 



I 



note that these frame -types would be produced by the matrix 
technique in the programmer's effort to deal systematically with the 
Important Intra-verbal connections \diich make up the subject-matter* 
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the BU<xr RtT’s Involved, on occasion we may wish to call for a 
statement of a specific RU. To exemplify; "What is the second law 
of thermodynamics?" 

9 . S. There is another inportant concept in the RU-EG frame- 
work. This is the concept of the "false exanple" vhich we symbolize 
like this; E5 (read "egg bar").^ The rationale for the EG is this; 
Opinion on the role of mistakes ^d errors in the course of programmed 
learning is divided. Some hold with Skinner that mistakes are to be 
avoided and that providing students with opportunities to make errors 
with a multiple -choice -type format is hazardous. Others, like Pressey 
and Crowder, feel that making mistakes and dealing with the conse- 
quences is a part of the way life is, and their devices, multiple- 
choice machine and scrambled textbook, reflect this approach. Others 
have even hypothesized an "optimal error rate" for programmed learning. 
Our philosophy is this: In addition to the rich intra-verbal as- 

sociations idiich the professionally competent person in some fields 
of learning has, he possesses an additional ability. This consists 
of his facility in detecting and correcting errors, even though he 
might never make those errors himself. Such "error -detecting" be- 
havior can be taught like any other behavior, i. e., by giving the 
student a chance to practice it and get reinforced for performing 
appropriately. Every field of knowledge has its own peculiar booby- 
traps and sources of confusion which become painfully well-known to 
the teacher. Rather than, have the student fall into these errors, and 
rely on uncertain punishment effects to correct his behavior, our 
technique has been to apprise -him. (the" EG)-,, and indicate that he is 
to correct it. By conside.,7lng error -detecting behavior as a part of 
the repertory of the e:q?ert we are trying to emulate, we can usually 
get students over rought spots in programs without actually having 
them emit the incorrect behavior. 

Examination of the protocol following a programmed learning 
session will help point up the sources of error, and subsequent 
programs can Incorporate EG's at the appropriate spot. An example 
of an EG: 
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In his hooevork a student had the following equation: 

3(5 + 2) * (3 + 5)(3 + 2). But you know this is a 
mistake. The correct equation is: 3(5 + 2) = f 

Ans. (3 3C 5) + (3 3C 2) 

Another use of the 1 g Is in getting the student to discriminate 
where and where not to use TV words. For example* 



[ Names of people and places ere termed proper nouns. For 

\ exainple^ "Chicago” is the name of a place ^ so it is a 

proper noun. "Tree" is not the name of a person or 
place so it is not a ! N 



s 

r 

! 
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To summarize the use of the EG: When inspection of the RU matrix 

or program protocols indicate error-likely spots, present the error 
labeled as such and permit the student to deal with the error idiile 
still receiving positive reinforcement (confirming feedback). 

As an illustration of varkous frame types, the Appendix to this 
report presents a revision of Skinner's sample physics program in 
RULES terms. This illustrative program is taken from an Initial 
paper by Hbmme and Glaser (Problems in Programming, Lloyd E. Hbmme 
and Robert Glaser, University of Pittsburgh. A paper presented at 
the meetings of the American Psychological Association in Cincinnati, 
Ohio, September, 1959. ) >*lch involved a brief discussion of the 
RUIEG notion. 



STEP 9. Using the numbered RU matrix as a prompt, assemble the frames 
of various types into a program . Among the problems to be faced at 
this step are the following: How many frames should be used for each 

RU? How should these ffames be distributed throughout the program? 

Our procedure is to lead off with a RU t EG EG frame, and terminate 
when the student can deal effectively with EG frames (end RU frames). 
In between, the general principle is to withdraw stimulus support 
"gradually", with the use of such frame -types as RU + EG, EG EG, 
and EG ^ as mentioned before. Actually, prior to the first admin- 
istration of the program, the optimal number and distribution of frame* 
types within a sequence can only be estimated. !Ibis leads to the next 

step. 

- 13 - 









IBPFJIgIBiiUU.. Ml I Jill I M II I M I II III II I II II II , I I '.''M.liWjjlggppjt ii'iMi lip Mjiniiiij P Ml I 









STEP 10. Give the program to students and ”ltem"analy2e” their 
responses to each fypme . Only after a frame -by-frame analysis of 
students' recorded responses to the program can the effectiveness 
of a program be assessed. If the students are dealing effectively 
with EG 's^ then the program has done Its Job and further 
modifications can be in the direction of malcing It more efficient 
without Interfering with Its effectiveness. If the student Is not 
producing the specified criterion behavior reliably, then steps must 
be taken to add more fl*ames, redistribute the frames, clear up am- 
biguities, and so on. One of the chief strengths of the programming 
technique Is Its provision for analyzing and revising material on a 
detailed empirical basis for such analysis and revision, 1. e., the 
responses of the learner. 

STEP 11. Revise the program on the basis of students' responses and 
comments . Students should be encouraged to record their comments and 
criticisms on particular frames as they proceed through the program. 

The most difficult phase consists of convincing the student that when 
he makes an error the blame is on the program and not on himself. This 
often takes some doii^, for most students typically enter the programming 
situation with a long and consistent history of having their faults and 
inadequacies pointed out to them. Near-errorless learning msy come as 
something of a shock to them. But the student zmist be convinced that 
he is the most important participant in the effort for systematic 
Isrprovement of the program. A£ter Initial administration of a program, 
a large proportion of the corrections will follow quite obviously from 
the nature of the errors made. Some will be a result of the sinq)le 
typographical errors; others will be produced by easily-corrected am- 
biguities in the text. Failure to come up with a correct tern or 
solution implies the necessity of additional prior strengthening of 
the concept involved. 

To pay lip service to the idea of the constantly revised program 
is one thing; to do the actual dog -work is another. Ihis is why it is 
risky to use any elaborate duplicating procedure early in the game. 

Frames written on separate index cards will probably get added to, 
removed, and edited. Programs prepared in ditto or mimeographed 
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page form mtiy be discarded and replaced by revised programs. But 
programs on long continuous tapes or nicely bound and printed are 
probably "locked in", and are quite likely to remain in that form. 

!Ihe point is that you are not writing a textbook; you are preparing 
a sequence \*lch will teach a specified behavior, and you don't want 
to go to press until this program has demonstrated that it can do its 
job. 



Precautions . In the process of program revision, there are a number 
of booby-traps. The first is the dager of over -prompting . If a 
student cannot come up with the word "entropy", the solution is not 
to give him "entrop_^" and have him guess the letter "y". With 
adequate prompting, s: jungle savage can get through a course in dif- 
ferential equations. It is easy to correct a program by adding 
prompts, re -administer 5.ng it, and find in g to one's delight that the 



errors drop practically to zero in the corrected frames. But the 
proof of the pudding is in the S's; only if the student can produce 



criterion behavior with minimal cueing has a revision succeeded. 

A second booby-trap consists of assuming that the student will 
respond appropriately to supplemental instructions, comments, and 
material \fblch appears on the frame but is not involved in the actual 
production of the response to that frame. For exaznple, do not pre- 
sent the quadratic formula and say "learn this". Do not print "read 
each frame carefully" on each item. Such techniques are probably not 
without effect, but they have a hard-to -control differential effect on 
students of varying compulsivity. A more realistic rule-of-thumb is: 
The student will attend only to those stimuli which are necessary to 



produce the correct response on any given frame . Exhortations to 
learn well and not to moke errors are tenuous sources of control of 
student behaviors; behavior important enough to be strengthened at 
all should be dealt with by evocation and reinforcement. 

The third booby-trap is some^at allied to booby-trap number two, 
and is intimately related to the RULES system Itself. It stems from 
the fact that as we formalize a system of program writing, as has 
been done in RULES, the responses of the student may fall under the 
control of the wrong set of stimuli. Many of us in our childhood had 
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exgeTlenoe with a "teaching machine" in i^ich the learner used a vire 
to connect a question^ e«8*i ^&t is the capital of Arizona?^ to 
e possible answers. ^ Joining the correct question and answer^ an 

r 

electrical circuit was made which rang a bell. The difficulty with 
such a device was that the learner soon learned the relative position 
between questions and answers^ and was able to give the correct answer 
Independently of the verbal materled idilch was on the device. An 
analogous danger exists in formalizations such as RUL&G. For example^ 
if we underlined all TF words (technical vocabulary) the first time 
they were presented and always called for the TV word later on the 
same frame, the production of this word might soon fall under control 
of the underline, rather than under control of the textual material 
present. 

On the other hand, forxaallzation is not without its benefits. We 
have some observational evidence that some sort of a "learning how-to- 
learn" phenomenon can take place, lhat is, once a student gets into 
the swing of the BU ■i’ EG -f EG formula, he begins to grasp new BU's 
t, very quickly because he can begin to anticipate how the presentation 

of the rule will be developed. If he finds that he is always required 
^ to state each RU at some later frame (as in a BU), this may lead him 

to attend more carefully to the BU as it is first presented. 

STEP 12. Repeat the administration and revision procedures until 
the program is producing criterion behavior reliably and efficiently . 
This last step concludes the presentation of the BULEG system of program 
construction. We trust that this early attempt to formalize and make 
explicit some of the rules of programming will be Judged by the some 
criterion we use to ascertain the effectiveness of a program, namely, 
by results. like a program, we hope that it will be edited, revised, 
supplemented, or discarded on the basis of the results it produces. In 
particular we would like to dispel any ispresslon that we are lr» 
revocably committed to a BUIEG approach, or that we feel BULEG will 
^ solve all programming problems, ^e system contains a large number of 

gaps and guesses. But in our own work we have found BULEG to be an 
enormously helpful prompting system. Previous to a semi -formalization 
of BULEG, our programming behavior could best be described as sort 







of an artistic manipulation of the subject matter. Also^ ve found it 
almost in^osslble to be of any help to people \rbo wished to learn to 
program^ beyond such nebulous advice as "Proceed carefully in small 
steps." i^ter werbegan to be more esg^llclt (and dogmatic) in the 
rules for program construction^ we noticed that our own frame produc- 
tion went up tremendously^ and we:’found that for the first time we 
could make some sensible and helpful suggestions to beginning program- 
mers. We have found the HUIEG system flexible enou^ to be applied to 
such diverse topics as statistics^ Hebrew^ Hungarian^ physics^ symbolic 
logic ^ and investment banking; and communicable enou^^ to be grasped 
eind employed by bright undergraduates. Finally^ our chief claim for 
the RUIF)G system is that it helps produce programs. Once a program 
has been constructed^ then it can be evaluated and modified on the basis 
of the responses of the learner and his attednment of the criterion 
behavior. 



* 
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This appendix contains an illustrative revision of Skinner »s 
sasjple physics program rewritten in Ruleg terms* The original 
program excerpt appeared in Science j 24 October 195®/ Vol* 128^ 

3330, 969 - 977 . !Ihe Ruleg revision appearing here was first described 
in a paper by Homme and Glaser ^dlicb appears in the book Teaching 
Machines and Programmed Learning by A. A. Luosdaine and R. Glaser, 
i960. National Education Association. 
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Skinner’s High School Physics Program Reconstructed 
According to the Ruleg System 



Class 



Sentence to be con 5 >leted 



Word to be 
supplied 



ru + eg 



eg 



ru + *eg 



eg 



eg 



ru 



ru + ru 



ru 4- i»u 



o 

ERIC 



1 . 



To “emit®* light means to “send, out** light. 
For exan^jle, the sun, a fluorescent tube, 
and a bonfire have in common that they all 
send out or light* 



emit 



2 . 



A firefly and an electric light bulb are 
alike in that they both send out or 
light. 



emit 



3. 



Any object which gives off light because 
it is hot is called an incandescent light 
source. Thus, a candle flame and the sun 
are alike in that they both are 
sources of light* 



incandescent 



4-. When a blacksmith heats a bar of iron \intil 
it glows and emits light, the iron bar has 
become a(n) source of li^t* 



5. A neon tube emits light but remains cool, 
Ikilike the ordinary electric li^t bulb, 
then, it is not an 
of light. 



6. An object is called incandescent when 



7* Xt has been found that an object, an iron 
bar, for exanple, will emit light if its 
ten^erature is raised above 800 degrees 
Celsius . Therefore, we say that above 

_ (temperature) ob- 



jects will become 



8. An electric light bulb produces light when 
the fine wire, technically called a fila- 
ment . inside the glass is heated to incan- 
descence. This means, then, that the fine 
wire or must exceed a tei» 5 )er- 

ature of about ^ Celsius to emit 

light. 



incandescent 



incandescent source 



it emits li^t be- 
cause it is hot 



800® Celsius 
incandescent 



filament 

800 















eg 



9 . 



ru + eg 
+ 



10 . 



ru 



r^ 

eg 



n. 



^\J 

ru 



12 . 



ru + eg 13« 



eg 



14. 



ru 



ru + eg 



e 

yc 



In an electric li^t bulb *when an electric 
current is passed throu^ the fine wire or 

, it becomes « because 

it is heated to a temperature above 800® filament 
Celsius. 



incandescent 



The hotter an incandescent li^ht source 
becomes, the greater the amount of light 
it emits. For exan?>le, an object heated 
to 900° Celsius would emit more light than 
the same object heated to 800° Celsius be- 
cause the light emitted de- 

of the object. 



pends on the 



amount 

tenqperature 



A nearly “dead” battery in a flashlight pro- 
duces only a feeble li^t in the bulb, be- 
cause the filament fails to get - ■ - 

enough to produce much light. 



hot 

heated 



An incandescent light source can be made 
non-incandescent by cooling it, since the 
amount of light emitted depends on the 
of the source. 



temperature 



The color of the light an incandescent 
object emits changes from red to white as 
the object’s temperature increases. Thus, 
when a blacksmith heats a piece of iron it 
first glows (color), and then 



as it gets hotter turns 



red 

wh3.te 



A weak flashlight battery produces only a 
dull (color) glow in the bulb’s 

filament and little light is emitted be- 
cause the of the filament is 



relatively low. 



15. Both the 



and 



of 



light emitted by an incandescent object 
depends on the teuperature. 



16% In “emitting light” an object changes or, 
technically, "converts” one kind of energy 
to another. For example, in a flashlight 
the electrical . supplied by the 

battery is changed or ed into heat 

and li^t. 



red 
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tenperature 
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amount 
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color 






1 

-I 

1 


energy 
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converted 
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